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United Space Alliance (USA) 

Who are we? 

- NASA's primary industry partner in human space 
operations, including the Space Shuttle and the 
International Space Station. 


- Three primary 

• Johnson Space 

• Marshall Space 

• Kennedy Space 

-About 10,000 employees 
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Predictive Maintenance Engineering Lab (PMEL) 


Who are we? 

* Predictive Maintenance for the Space Shuttle Program is 
based out of KSC and utilizes state-of-the-art technologies 
to monitor and trend equipment health. 

Bearing Failure Analysis Thermography 

Fluid Analysis Ultrasonic Noise Detection 

Motor Circuit Evaluation Vibration Analysis 

* Equipment condition trends are used to validate 
preventive maintenance cycles, reduce unplanned 
maintenance downtime, and improve machine reliability. 

* PMEL Engineers perform acceptance and validation 
testing for new and rebuilt machinery thereby establishing 
the baseline for equipment health. 



United Spmcm AMantm 


The predictive maintenance engineering group. Formed in 1989 using 
Vibration analysis only, added Fluid and Thermography in 1992. MCA, 
laser alignment and ultrasonic noise detection followed soon after. 
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Shuttle Landing Facility (SLF) 
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KSC is overlaid with 140,000 acres of the Merritt Island National Wildlife 
Refuge. 

Warn of the acronyms 

Orbiter landings at the Kennedy Space Center are made on one of the 
largest runways in the world. 

This aerial shot gives some idea of what approaching aircraft 
see as they reach the KSC shuttle runway. It's not quite this 
angle for a returning shuttle, which drops at a more severe 
incline and incredibly high speed on its way in. Kennedy Space 
Center. 

The runway is 1 5,000 feet long and 300 feet wide with a 1 ,000-foot safety 
overrun at each end. In general terms, the runway is roughly twice as 
long and twice as wide as average commercial runways, although a 
number of domestic and foreign airports have landing strips far exceeding 
average dimensions. 

The Mate/Demate device (MDD) used to raise and lower the orbiter from 
its 747 carrier aircraft during ferry operations. The open-truss steel 
structure is equipped with hoists, adapters and movable platforms for 
access to certain orbiter components and equipment. 


MDD Hoists that lift the Orbiter, MCA on Portable Purge Units (PPU), and 
Ground Coolant Units (GCU). 

MCA data will be done on DC motors, now that we have the capability. 

The landing facility is linked with tR0a0r0iter Processing Facility by a 2- 
mile tow-way. 
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Orbiter Processing Facility (OPF) 



After landing, the orbiter is towed to the Orbiter Processing Facility 
(OPF). The orbiter then undergoes safing procedures in the OPF which 
include removing residual hypergolics. Then the orbiter's previous 
mission payloads are removed and the vehicle is fully inspected, tested, 
and refurbished for its next mission. 

Hyper Exhaust fans, All ECS equipment, Orbiter Hydraulics, Bridge 
Buckets, and HVAC equipment. 

Image Credit: NASA 
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Vehicle Assembly Building (VAB) 
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Image Credit: NASA 

The Vehicle Assembly Building (VAB) is one of the largest buildings in 
the world. It was originally built for assembly of Apollo/Saturn vehicles 
and was later modified to support Space Shuttle operations. 

Lifting Devices: 71 cranes; two 227 metric ton (250 ton) bridge cranes. 

During Space shuttle build-up operations inside the VAB, integrated SRB 
(How Many segments?) segments are hoisted onto a Mobile Launcher 
Platform and mated together to form two complete SRB s. The ET, after 
arrival by barge, is inspected and checked out and then attached to the 
SRBs already in place. The orbiter is then towed over from the Orbiter 
Processing Facility to the VAB transfer aisle. 
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Going Vertical in the VAB 
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Image Credit: NASA 

The Orbiter is raised over the 16 th floor to a vertical position, lowered onto 
the Mobile Launcher Platform and then mated to the rest of the stack. 
When assembly and checkout is complete, the crawler-transporter enters 
the High Bay, picks up the platform and assembled shuttle vehicle and 
carries them to the launch pad. 

List equipment PMEL tests in the VAB 

HVAC equipment, Doors, Platform hoists, Mini PPU’s. The large Cranes 
are DC drives and we will begin testing them now that we have the 
technology. 
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Launch Pads 39 A/B 



Launch Complex 39's Pad A and Pad B were originally designed to 
support the Apollo program and were modified for Space Shuttle launch 
operations. Pads 39-A and 39-B are virtually identical and roughly 
octagonal in shape. 


Propellant servicing of the Space Shuttle's reaction control systems, the 
booster auxiliary power units and the external tank is performed at the 
launch pad. Fuel lines lead from the propellant storage facilities to the 
pad structure and onward to the umbilical connections. These facilities 
include the hypergolic storage and distribution facilities. 

The orbiter's Orbital Maneuvering Subsystem (OMS) and Reaction 
Control System (RCS) engines use monomethyl hydrazine (MMH) as fuel 
and nitrogen tetroxide (N204) as the oxidizer. These toxic fluids can be 
stored at ambient temperatures. Being hypergolic they ignite on contact 
with each other. Therefore, they are stored in well-separated locations, at 
the southwest and southeast corners of the pads. 

List Equipment PMEL tests at the Pads 


All Environmental Control System (ECS) equipment, LOX Pump Motors 
(hardwired SKF vibration system), OMS Hypergolic Pumps, and Heating 
Ventilation Air Conditioning (HVAC). 

National Aeronautics and Space Administration 
John F. Kennedy Space Center 
Kennedy Space Center, Florida 32899 
FOR RELEASE: 03/01/2006 
PHOTO NO: KSC-06PD-0390 
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STS-118 
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The Rotating Service Structure (RSS), moves around the shuttle for 
processing at the Pad and for protection. It aslo has a six story 100K 
class clean room, used for payload processing. The Fixed Service 
Structure (FSS) was orgianally used for the Apollo program, it was called 
the Launch Umbilical Tower (LUT) 
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STS-117 June 2007 
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Shuttle takes off ~4 million miles later it lands (Edwards or KSC) taken to 
OPF and processing starts again. 


Mission: Space Station Assembly Building - 13A 
Primary Payload: S3/S4 Truss 
Space Shuttle: Atlantis 
Launch Pad: 39A 

Launched: June 8, 2007 - 7:38:04 p.m. EDT 
Landed: June 22, 2007 - 3:49 p.m. EDT 

Mission Duration: 13 days, 20 hours, 12 minutes and 44 seconds 
Inclination/Altitude: 51.6 degrees/122 nautical miles 


STS-117: Building on Experience 

Space Shuttle Atlantis lifted off on mission STS-117 after a perfect 
countdown June 8, 2007, marking the first launch from Pad 39A at 
Kennedy Space Center in more than four years. During the 14-day 
mission, the astronauts conducted a total of four spacewalks to install 
and activate the S3/S4 truss segment, retract the P6 solar array and 
repair an out-of-position thermal blanket on the left orbital maneuvering 
system pod. 
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Test Set-up and Pad A pumps 
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Not a typical set-up for these pumps, Baker loaned another tester so both 
pumps could be tested at the same time. 

(Right picture) Pump #1 is at top center, discharge of this pump is rotated 
90 deg so it’s a straight shot to Pump#2. 
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Propellant Servicing 
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SCAPE operations for S00024; Pad is cleared, only personnel in SCAPE 
suits are allowed on Pad or at the Farms. Oxidizer is loaded then Fuel. 

Here is a synopsis of the Hyper Load. 

Pump #1 is turned on and pump #2 is turned on 30 seconds later. Output 
pressure DS of pump #2 runs between 315-330 psi. 

•Flow is established up the tower and re-circulates back to 
the storage area. 

•Once flow is detected at the skid inlets, flow is established 
in the skids and a leak checks of the skids are performed. 

•Flow in the skids is maintained throughout the loading 
process. 

•The FWD and AFT tanks are loaded in parallel. 

•In the AFT, the OMS tanks are loaded first. Left OMS is 
loaded and then the Right OMS. (Fuel loads the right first 
then the left) 

•Once OMS tanks are loaded the AFT RCS tanks are 
loaded. Left first then right. (Fuel loads the right first then 
the left) 

•When both the FWD and AFT tanks are loaded and 
everyone is satisfied with the numbers, flow termination is 
initiated. 

•Pumps are shut down and propellant freefalls back to the 
storage area. Page 12 

•Blow downs are performed from the tower to the Storage 


Pad Hyper Pump Testing 


* PM cycle is very short, Oxidizer pumps must be inspected every 100-hours. 
Fuel pumps are on 300-Hour schedule. 

* What are likely failure or historical failure modes 

- Journal bearing wear 

• Rotor stator contact and 

* Breaching jacket on stator 

* PMEL began testing 

- Vibration test run in 1998 written into procedures in 2005. 

- Explorer first test run, August 2006. 

* Operational Constraints 

- S.C.A.P.E. hazardous operations 

- All other processing at the Pad stops during S00024 operations. 

* Test Setup Challenges 

- Monitor one pump at a time 

- timed loop acquisition 
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Two tester are required so both pumps can be tested at the same time. 

Since they have to capture data when no one is around, the Loop 
Acquisition function on the Explorer software is the only option available. 
They typically will take one run every 10 to 20 minutes. Propellant 
servicing requires three days of 24 hour operation, the Oxidizer pumps 
may operate for more than 7 hours at a time. 
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CRANE CHEMPUMP Series G 



For normal clear fluid applications, Series G Chempumps are cooled and lubricated 
by the fluid being pumped, which flows through the circulating tube, into the rear of 
the rotor chamber, across the rear bearing, rotor, and front bearing, and then back 
into the main pumped stream through small holes provided in the rear of the 
impeller in the suction area. 


USA 

Pump #1 discharge is rotated 90 deg so it’s a straight shot to Pump#2. 

Rotor and stator are jacketed in Stainless steel. Primary failure mode 
concern; rotor stator contact and breaching jacket on stator which would 
result in an explosion and release of toxic propellants. 

http://www.chempump.com/chempump_products/g_series/index.html 

Information from Crane website. 
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Journal Bearings 


Normal Bearing 



Showing Excessive Clearance 
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These pictures can be used to illustrate how the oil whirl can exist in a 
journal bearing. 

Left: shows normal bearing but it is somewhat deceptive because 
everything is symmetrical, usually this is not the case. Even in new 
bearings a high pressure wedge of oil forms depending on rotation where 
the oil is pressed between the shaft and bearing, a low pressure area 
forms on the opposite side of this. A bearing within tolerance will come to 
some position of equilibrium and usually stay there due to the pre-loading 
on the shaft. 


Oil whirl typically shows up as a sub-synchronous vibration of the shaft 
speed (35% to 48% of RPM frequency) 

The causes of oil whirl are usually poor bearing design, and bearing 
wear, but it can also be present on lightly loaded machines or a change in 
the fluid viscosity or machine alignment are other possibilities. 

You will notice on the Pad Hyper pumps the load never dropped below 
60%, so we can eliminate this possibility. 
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Possible Causes of Fluid Whirl 


•Poor bearing design 

•Not an issue with these bearings 

•Lightly loaded machine 

•Not an issue during normal operations 

•Change in fluid viscosity 

•Not an issue during this operation, fluid is sampled before 
and after operation 

•Bearing wear 

•Likely cause in these pumps 


USA 

These pictures can be used to illustrate how the oil whirl can exist in a 
journal bearing. 

Left: shows normal bearing but it is somewhat deceptive because 
everything is symmetrical, usually this is not the case. Even in new 
bearings a high pressure wedge of oil forms depending on rotation where 
the oil is pressed between the shaft and bearing, a low pressure area 
forms on the opposite side of this. A bearing within tolerance will come to 
some position of equilibrium and usually stay there due to the pre-loadinq 
on the shaft. 


Oil whirl typically shows up as a sub-synchronous vibration of the shaft 
speed (35% to 48% of RPM frequency) 

The causes of oil whirl are usually poor bearing design, and bearing 
wear, but it can also be present on lightly loaded machines or a change in 
the fluid viscosity or machine alignment are other possibilities. 

You will notice on the Pad Hyper pumps the load never dropped below 
60%, so we can eliminate this possibility. 
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Pad B Ox #1 


11-19-06 02:38:12 PM 





NEMA Volt Unbalance C%I 
NEMA Derating Factor 

Speed IRPMJ [3512.5 

Torque IfMb] [5S1 

Load [HP] [23.3 

% Load [64.3 

% Efficiency [89.4 


Torque -Spetd 


. . 

Autophase 

Yes 
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The first test we will show was taken on Pad B Oxidizer #2 in November 
of 2006. 


Propellant servicing requires three days of around the clock operation, 
the Oxidizer pumps may operate for more than 7 hours at a time, this last 
time Oxidizer load took more the two days. 

This is just one of 23 runs captured on Pump #1 in November of 2006. 

No problems noted on the front panel, except the speed is off but that is 
not apparent right now. 
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Pad B Ox #1 11-19-06 02:38:12 PM 
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SC4 


Torque ripple looks ok at ~5%, rotor bar sidebands appear to be ok. Note 
speed capture is incorrect, this may be due to the very high slip of these 
motors. Current spectrum (0-60 Hz) looks good, and all other electrical 
data looks nominal. 
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Rotor Speed = Svnc - YFp'p * Svnct / 1201 

3600 - (3.29 x 3600)/120= 3501.3 RPM 

Fluid whirl should appear at 35% to 45% 
of Mechanical Frequency. 

3501.3 / 60 = 58.35 Hz 
58.35 x 0.35 = 20.42 Hz 
TO 

V 

58.35 x0.48 = 28.00 Hz 




Pad B Ox #1 


11-19-06 02:38:12 PM 
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Linear plot of Torque spectrum taken at the same time. The peak at 3.29 
Hz is the Fpp (Frequency Pole Pass). Rotor speed can be calculated 
from this peak very accurately using the following formula ... 

Sync - (Fpp * Sync) / 120) so 3600 - (3.29 * 3600)/120= 3501 .3 RPM. 

Since the primary failure mode of these pumps are rotor stator contact by 
excessive bearing clearance. 

Fluid whirl should appear as a sub-synchronous peak of the mechanical 
frequency. 

Mechanical frequency calculated by RPM/60 or 3501.3/ 60 = 58.35 Hz 
(may or may no show in torque spectrum indication of imbalance). 

Oil Whirl calculated at 35% to 48% of Mechanical Frequency 

Where 0.35 * 58.35 = 20.42 Hz to 0.48 * 58.35 = 28.00 Hz 

Based on the calculations and the spectrum, there are no indications of 
fluid whirl. 
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Pad B Ox #1 8-12-06 Vibration Spectrum 
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Summary of Ox #1 analysis 


• Explorer and Vibration analysis data are in agreement 

• Pump healthy no fluid whirl 

• Recommendation continue normal operations until 100 
hour planned maintenance cycle 


us a 


So to review on Pad B Oxidizer #1 the Explorer results were validated by 
vibration analysis. 

The pump was determined to be healthy and we recommended to 
continue normal operations. 

Unfortunately there is still has a 100 Hour disassembly rule for these 
pumps. 
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Pad B Ox #2 8-13-06 03:50:14 AM 
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SC7 

This is one of 34 runs captured in Aug 2006 on Pump #2. Again there 
are no apparent anomalies. 

Voltage looks good, balanced current, good power factor, speed is 
believable (for now), % load is good, everything looks great. 
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Pad B Ox #2 


8-13-06 


03:50:14 AM 
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SC8 

Torque ripple is good at <5% actually a little lower than Pump #1 , rotor 
side bands greater than 45 dB down also shows in current spectrum at 
56.27 Hz (missed the speed again). There does appear to be an 
unknown peak in the current spectrum at approximately 38.02 Hz, all 
other electrical data is nominal. 
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Pad B Ox #2 


8-13-06 


03:50:14 AM 
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~SC9 “ 

Linear plot of Torque spectrum. Fpp at 3.75 Hz = 3487.5 RPM (Note 
Pump 1 Fpp level was ~.07 N-m). 

Mechanical frequency 3487.5/60 = 58.12 Hz 

Obvious peak at 22.00 Hz ( 22.00/58.12 = 0.378) or 38%. This is 
indicative of fluid whirl. 

Interestingly enough, there is also a peak at approx. 44 Hz. It may be an 
fluid whirl severity indicator which would be useful in determining 
remaining useful life. 

Side note: 60 — 22.00 = 38 Remember 38.02 Hz peak in current 
spectrum? Strange that it appears in all 0x2 data but not in Oxl. 
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280. 

9H/I7/2OO0 1:59:44619 PM 

Y: .9409334 


X: ?1.3 75 


Pad B Ox #2 


8-12-06 Vibration Spectrum 


08 Motor Bearing 
091 

Horizontal 


Band Overall' .0677867 
Peak Amp: .0409334 
Speed: 57.5 
Sync .6115845 
SubSync: .0518549 
NonSync: 3)418319 
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Vibration was taken the night before and also shows fluid whirl @ 21.87 
Hz. 
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Summary of Ox #2 MCA analysis 


• Vibration data indicated fluid whirl 

• Explorer torque spectrum indicates possible fluid whirl 

- Possible severity indicator peak at 2x fluid whirl 

• Unknown peak at 38Hz in current spectrum may be an 
alternative method of detecting whirl. (60 - 38 = 22) 

• Eccentricity peaks at mechanical frequency times number 
of rotor bars is possible 

— Fmax was set to low to capture eccentricity peaks 
during this operation 


USA 

Since this was essentially the first time fluid whirl was seen with the 
explorer validation was needed. This included testing at Pad A and 
inspection of Pad B pump #2. 
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Pad B Ox #2 pump assembly 
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Pad B Ox#2 pump disassembled in March 2007. Clockwise from Left 
side; Rotor-impeller- and flow inducer, suction side of impeller, flow 
inducer, rear bearing housing, and the stator core. 
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Front Journal Ox #1 
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Pad B 0x#1 front journal 


~>Q 
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Rear Journal Ox #1 



Page 29 


29 





Front Journal 0x2 



USA 

Pad B Ox #2 Front journal. 
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Rear Journal 0x2 



Pad B Ox #2 Rear journal. 
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Bearing and Journal Dimensions (Ox 1 and 0x2) 


Manufactures Specifications 

Ox Pump#1 

Ox Pump #2 

Front 

Rear 

Front 

Rear 

Journal 

oo 

■ 

CO 

CO 

oo 

m 

1.183" 

1 1.184" 

1.184" 

1.184"! 

Bearing 

1.1880" -1.1890" 

1.189"| 

p19T 

1.193" 

■ 

Clear 

.0048" ■ .0057" 

0.006" 

0.007" 

0.009" | 


Note: Manufacture maximum diameterical clearance is 0 , 0 ) 4 " 1 


USA 

Looking at the table Oxidizer #1 front and rear journal and the front 
bearing are within new specifications. The rear bearing is slightly out of 
spec., but both could have been used and a considerable amount of life 
has been wasted. 

The same is true of the Front bearing on Oxidizer #2 but the rear bearing 
has considerable wear and is significantly out of tolerance. The 0.010” 
over MAX tolerance is believed to be why oil whirl was observed on Ox 
pump #2. 
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Root Cause of Pad B Ox Pump #2 Failure 



Housing flange was warped about 
0.020” causing the rear journal to 
contact the housing thereby 
resulting in excessive wear of the 
bearing. 



The root cause of the rotor rub was suspected to be the rear bearing 
housing flange. It was found to be warped ~0.020”. 
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Summary Ox 1 and 2 


•Pad B Ox 2 was disassembled first and showed significant 
wear on the rear bearing and also had significant rotor rub 

•If condition based maintenance were used it would have been possible to 
salvage the rotor and bearing, if the pump was removed from service earlier 


•Pad B Ox 1 disassembly showed no abnormal wear 

•With condition monitoring this pump could have remained in service and 
not been disassembled 

•This pump had considerable remaining useful life (RUL) which was wasted 
when it was disassembled 

•Re-assembly may introduce new problems into this pump 


USA 


•(NOTE first bullet: CBM was not being consistently being performed 
when pump first started) 
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Pad A Ox #2 


01-27-07 5:32:26 PM 



460.6 


VFD Details 


— 


d|345Q V o»s [460 Elec. Model |defaufl elec ^ ^ 


Machine \oX 2 

. .. '' ■.'* '• 


VU>| . Mo 




Volt 


|460.6 |460.6 

(302 (29.4 


Avg Froq [Hz] fioo 

NEMA Volt Unbalance [%1 


0.46 


NEMA Derating Factor 


Torque - Speed 


Speed 

Torque 

Load 


JRPM] 

[ft-lb] 

[HP] 


% Load 
% Efficiency 


Torque 




Autophase 

Yes 




Sensors 

Portable-^ 

EP-l , 


No 


01/Z7/2W7 11. 49.21 PM elec 
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This is one of 17 runs captured in January 2007 at Pad A Pump #2. 
Again no apparent anomalies. 
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Pad A Ox #2 


01-27-07 5:32:26 PM 



USA 

SC12 


Torque ripple is good at <4%, rotor side bands greater than 45 dB down 
also shows in current spectrum at 56.16 Hz (speed captured correctlyW) 
all other electrical data is nominal. 
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Pad A Ox #2 


01-27-07 5:32:26 PM 



SCI 3 

Torque spectrum looks nominal. No abnormal peaks. 


USA 
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Pad A Ox #2 01-27-07 Vibration Spectrum 
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SC13 


Vibration analysis reported fluid whirl in their report. This was not 
corroborated with the Explorer, which indicated the sub harmonic peak 
was a voltage distribution problem, there are SCR heaters on the bus 
which feeds these motors and the heaters were operating during this 
period. 

Advanced data analysis may be required to determine if erroneous 
vibration data exists. By exporting the raw data torque, voltage, and 
current), and plotting it was determined to be a voltage issue not fluid 
whirl. 
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Pad A Ox #2 


01-27-07 


Harmonics Run 5 
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Harmonic data does not reveal problems with voltage, because there 
were sub-harmonics on the line not harmonics the Crest Factor for 
Current does increase when these sub harmonics appear but not the 
Crest Factor for Voltage. 
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Preventative Maintenance (PM) 


* Preventative Maintenance: time-based tasks including inspection, 
service and/or replacement. 

• Established cycle based on average statistical data, anticipated lifetime, 
or manageable impact from temporary loss of asset. 


• PM may be invasive, requiring an outage and disassembly for visual 
inspection and/or overhaul/replacement regardless of condition. 

• The intervals between specific Preventive Maintenance tasks are based 
on average life or risk assessment. 


• A PM program can be cost effective when equipment operation is 
consistent, average life is predictable within a reasonable span, 
failures are well understood and manageable, and useful failure 
statistics are available. 


us£ 


In review, 

PM can be an effective maintenance practice IF a there is a sufficient 
population to base failure data on. 
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Predictive (PdM) or Condition Based Maintenance (CBM) 


• PdM/CBM: Maintenance action based on actual condition 
obtained from in-situ, non-invasive tests, operating and 
condition measurements. 

• Condition Based Maintenance has proven capable of identifying 
faults early enough to minimize the impact of: 

• operational interruptions 

• avoid expensive failures 

• including collateral damage 

• significantly reduce the cost of maintenance. 


* Some potential failures, such as fatigue, are not easily 
detected with condition measurements. 


USA 

CBM makes decisions on actual condition not statistical data. 
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Criteria for Applying CBM 
Preliminary Questions: 

Does the technology monitor a specific failure mode? 

Is it probable that this failure mode will occur during the lifetime 
of the equipment? 

Application Criteria: 

Changes in specific equipment parameters must correlate to 
corresponding failure mode. 

The parameter must be accurately and consistently measurable. 

There must be enough time between potential and functional failure to 
allow for corrective action. 

The monitoring interval must be shorter then the P-F interval. 

Multiple monitoring points during the P-F interval increases confidence in the 
diagnosis. 


USA 

1 , Yes, vibration analysis and MCA can both identify the primary failure 
mode on these pumps 

2. It has happened before, (1998). 
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P-F curve 


Point where 
failure starts 
to occur 


Vibration 

Analysis 

Detection 


Oil 

Analysis 

Detection 


Temperature 

Detection 


Audible 




The P-F curve is used to illustrate where a failure starts to occur in the 
life of a piece of equipment and where each technology could possibly 
detect that failure. I don’t like this curve too much because there is no 
one best analysis technique and it obvious, John Moubray came up with 
this before MCA was available... I doubt that but he may not have been 
aware of it’s power. 


Page 43 


P-F curve (modified) 
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Now this looks better. But if you think about ESA has actually found a 
fault before it has started to degrade the system, remember the Pad A 
Heaters? If this problem is not addressed then problems could escalate. 

But now the problem is trying to determine which technology, and at what 
frequency you should monitor every piece of equipment, 

If the time between the failure starting point and the functional failure is 
small then you just aren’t going to be able to catch the failure (typical of 
fatigue fractures). 

But if we continue to monitor the Pad Hyper pumps every time the 
operate then we should not miss a failure, unless the start of the failure to 
complete failure is very fast. 

Now journal bearings historically do not wear out very fast and there 
should be plenty of time to predict a problem with this failure mode. 
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On-Line Monitoring 


• Objective: Improve on-line monitoring of Oxidizer and 
Fuel motors during S00024 operations 

• Need: Because bearing wear out can lead to catastrophic 
failure. 

• Impact: provide early warning of impending failures which 
can be used for tactical decision making. With Explorer 
and Vibration data subtle changes can be trended and 
disaster can be avoided. 

* Maintenance decisions can be optimized by including condition 
based information to determine overhaul and planned 
replacement schedules. 

* Operational decisions can be improved by understanding and 
relating machinery condition to operational load and predicted 
time to failure. 

* The value of these On Line Monitoring techniques will be 
increased availability and reduced maintenance costs. 
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One objective of this presentation is to convince the decision makers that 
a CBM approach can detect the primary failure mode of these pumps and 
this could extend the lifetime and reduce total life-cycle costs 
considerably on these pumps. 
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Fault Detection 


• Issues: What fault modes are detectable with currently 
installed instrumentation? 

• Voltage/Current balance 

• Loss of power 

• What can be detected with Vibration Analysis and 
Explorer? 

• Rotor rub 

• Whirl (a.k.a. Oil Whirl) 

• Impeller issues 

• Resonance issues 

• Rotor faults (Porosity, broken bars, etc...) 

• Eccentricity (Static and Dynamic) 

• Electrical Faults and other hidden problems (power quality) 


USA 

1 1 Voltage and current are now monitored but this is sampled only a few 
time per second. One failure mode this may be able to detect would be 
shorted turns on one phase if current imbalance became noticeable. 

2 Combining Vibration and MCA would be able to detect the most serious 
failure modes rotor rub. 


Electrical Faults and other hidden problems Pad A heater operations. 
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Summary 


* Pad B Ox #1 Vibration analysis and Explorer data determined pump 
was in satisfactory condition 


• Pad B Ox #2 Vibration and Explorer detected fluid whirl, however there 
were no severity levels set so pump was run until 100 hour mandatory 
inspection 

- If alarm levels had been set and recommendations made to inspect this 
pump earlier the rotor and bearings may have been saved 


* Pad A Ox 2 Vibration analysis suggested there was fluid whirl on this 
pump, however Explorer determined this was not fluid whirl 

• The problem was determined to be voltage sub-harmonics causing 
excessive vibration which appeared as fluid whirl to vibration analyst 


USA 
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United Space Alliance (USA) 


Who are we? 

- NASA's primary industry partner in human space 
operations, including the Space Shuttle and the 
International Space Station. 

- Three primary locations: 

• Johnson Space Center, TX 

• Marshall Space Flight Center, AL 


* 


• Kennedy Space Center, FL 
- About 10,000 employees 


S 



Predictive Maintenance Engineering Lab (PMEL) 


Who are we? 

• Predictive Maintenance for the Space Shuttle Program is 
based out of KSC and utilizes state-of-the-art technologies 
to monitor and trend equipment health. 

Bearing Failure Analysis Thermography 

Fluid Analysis Ultrasonic Noise Detection 

Motor Circuit Evaluation Vibration Analysis 

• Equipment condition trends are used to validate 
preventive maintenance cycles, reduce unplanned 
maintenance downtime, and improve machine reliability. 

• PMEL Engineers perform acceptance and validation 
testing for new and rebuilt machinery thereby establishing 
the baseline for equipment health. 
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Pad Hyper Pump Testing 


* PM cycle is very short. Oxidizer pumps must be inspected every 100-hours. 
Fuel pumps are on 300 -Hour schedule 

* What are likely failure or historical failure modes 

- Journal bearing wear 

* Rotor stator contact and 

* Breaching jacket on stator 

* PMEL began testing 

- Vibration test run in 1998 written into procedures in 2005. 

- Explorer first test run. August 2006 

* Operational Constraints 

- S.C.A.P.E. hazardous operations 

- all other processing at the Pad stops during S00024 operations. 

* Test Setup Challenges 

- Monitor one pump at a time 

- timed loop acquisition 




CRANE CHEMPUMP Series G 



For normal clear fluid applications. Series G C hem pumps are cooled and lubricated 
by the fluid being pumped, which flows through the circulating tube, into the rear of 
the rotor chamber, across the rear bearing, rotor, and front bearing, and then back 
Into the main pumped stream through small holes provided in the rear of the 
impeller in the suction area. 






Causes of Fluid Whirl 

•Poor bearing design 
•Lightly loaded machine 
•Change in fluid viscosity 
•Bearing wear 
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Pad B Ox #1 8-12-06 Vibration Spectrum 



Summary of Ox #1 analysis 

• Explorer and Vibration analysis data are in agreement 

• Pump healthy no fluid whirl 

• Recommendation continue normal operations until 100 
planned maintenance cycle 
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Pad BOx #2 8-13-06 03:50:14 AM 



3600 • (3.75 x 3600)/120» 3467.5 RPM 
3467 .5 1 60 « 56.12 Hz 

x ZZ r— 54 12 “ ° “ “ 2034 

!ir to 

56.12 x 0.46 ■ 27.90 Hz 




Summary of Ox #2 MCA analysis 


• Vibration data indicated fluid whirl 

• Explorer torque spectrum indicates possible fluid whirl 

- Possible severity indicator peak at 2x fluid whirl 

• Unknown peak at 38Hz in current spectrum may be an 
alternative method of detecting whirl. (60 - 38 = 22) 

• Eccentricity peaks at mechanical frequency times number 
of rotor bars is possible 

- Fmax was set to low to capture eccentricity peaks 



Pad B Ox #2 pump assembly 
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Bearing and Journal Dimensions (Ox 1 and 0x2) 


Manufactures Specifications 

Ox Pump #1 

Ox Pump #2 

Front 

Rear 

Front 

Rear 


Bearing 

1.1880" •1.1890" 

1.189" 

1.191" 

1.193" 

1.208" 

Clear 

.0048" -.0057" 

0.006" 

0.007" 

0.009" 

0.024" 


Note: Manufactures maximum diameterical clearance is 0.014' 



Root Cause of Failure 



Housing flange was warped about 
0.020” causing the rear journal to 
contact the housing thereby resulting 
in excessive wear of the bearing. 


Summary Ox 1 and 2 


•Pad B Ox 2 was disassembled and showed significant wear 
on the rear bearing and also had significant rotor rub 

•If condition based maintenance were used it would have been possible to 
salvage the rotor and bearing. If the pump was removed from service earlier 


•Pad B Ox 1 disassembly showed no abnormal wear 

•With condition monitoring this pump could have remained in service and 
not been disassembled 

•This pump had considerable remaining useful life (RUL) which was wasted 
when it was disassembled 

•Re-assembly may introduce new problems into this pump 
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Preventative Maintenance (PM) 


Predictive (PdM) or Condition Based Maintenance (CBM) 

• Preventative Maintenance: time-based tasks includina inspection. 


• PdM/CBM: Maintenance action based on actual condition 

service and/or replacement. 


obtained from in*situ, non-invasive tests, operating and 

• Established cycle based on average statistical data, anticipated lifetime. 


condition measurements. 

or manageable impact from temporary loss of asset. 

• PM may be invasive, requiring an outage and disassembly for visual 


• Condition Based Maintenance has proven capable of identifying 
faults early enough to minimize the impact of: 

inspection and/or overhaul/replacement regardless of condition. 

• The intervals between specific Preventive Maintenance tasks are based 


* operational interruptions 

on average life or risk assessment 


- avoid expensive failures 

• A PM program can be cost effective when equipment operation is 


• including collateral damage 

• significantly reduce the cost of maintenance. 

consistent, average life is predictable within a reasonable span, 
failures are well understood and manageable, and useful failure 


• Some potential failures, such as fatigue, are not easily 

statistics are available. 


detected with condition measurements. 
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Criteria for Applying CBM 
Preliminary Questions: 

Does the technology monitor a specific failure mode? 

Is H probable that this failure mode will occur during the lifetime 
of the equipment? 

Application Criteria: 

Changes in specific equipment parameters must correlate to 
corresponding failure mode. 

The parameter must be accurately and consistently measurable. 

There must be enough time between potential and functional failure to 
allow for corrective action. 

The monitoring interval must be shorter then the P-F interval. 

Multiple monrtonng pomts during the P-F interval increases confidence m the 
diagnosis 





On-Line Monitoring 


Objective: Improve on-line monitoring of Oxidizer and Fuel 
motors during S00024 operations. 

Need: Because bearing wear out can lead to catastrophic 
failure. 

Impact: provide early warning of impending failures which 
can be used for tactical decision making. With Explorer 
and Vibration data subtle changes can be trended and 
disaster can be avoided. 

• Maintenance decisions can be optimized by including condition 
based information to determine overhaul and planned replacement 
schedules. 

• Operational decisions can be improved by understanding and 
relating machinery condition to operational load and predicted 
time to failure. 

• The value of these On Line Monitoring techniques will be 
increased availability and reduced maintenance costs. 



USA 


Fault Detection 


Summary 

• Issues: What fault modes are detectable with currently 
installed instrumentation? 

• Voltage/Current balance 


• Pad B Ox f 1 Vibration analysis and Explorer data determined pump was in 
satisfactory condition 

• Pad B Ox f 2 Vibration and Explorer detected fluid whirl, however there were 
no severity levels set so pump was run until 100 hour mandatory inspection 

• Loss of power 

• What can be detected with Vibration Analysis and 
Explorer? 

• Rotor rub 

• Whirl (a.k.a. Oil Whirl) 


- If alarm levels had been set and recommendations made to inspect this 
pump earlier the rotor and bearings may have been saved 
• Pad A Ox 2 Vibration analysis suggested there was fluid whirl on this pump, 
however Explorer determined this was not fluid whirl, the problem was 
determined to be voltage sub-harmonics causing excessive vibration which 
appeared as fluid whirl to vibration analyst 

• Impeller issues 

• Resonance issues 

• Rotor faults (Porosity, broken bars, etc...) 

• Eccentricity (Static and Dynamic) 



* Electrical Faults and other hidden problems (power quality) 





— — ssssasr- = 
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Summary of Fluid Whirl Detection 


• Explorer can detect it 

- Like Vibration the fluid whirl peak ranges from approximately 35 to 48 
percent of the mechanical frequency or RPM 

- Potential of severity by comparison with Fpp levels or if 2x peak is 
present 

- Eccentricity peaks at mechanical frequency times number of rotor bars is 
possible 

* Fmax should have been set higher to capture eccentricity peaks 

• Vibration should be cautious with interpretation, could be voltage issues 

• Advance data analysis may be required 

• Use of multiple technologies is helpful to validate results 

- Need to correlate Vibration and Explorer data as soon as operation is 
completed and data retrieved and analyzed 

- Process data (flow rate, pressure, temperature, etc.) could also be 
recorded and trended to help validate data 



Questions??? 
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